Scripts - HRRR - Check that all requested variables are in the grib file #35
Scripts - HRRR - Check that all requested variables are in the grib file #35brentwilder wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a post-hoc integrity check to the HRRR download script that confirms every variable listed in HRRR_VARS is present in an existing grib file. When the check fails, the file is removed so the normal archive-fallback path will redownload it from a different source on the next run.
Changes:
- New
check_file_is_validhelper runswgrib2 -s, parses fields 4-5, and grep-matches eachHRRR_VARSentry. check_file_existencenow invokes the validator on any non-empty file on disk and returns 3 (triggering the alt-archive pathway) when validation fails.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| check_file_is_valid() { | ||
| local file=$1 | ||
| local metadata | ||
| local var | ||
| metadata=$(wgrib2 "${file}" -s 2>/dev/null | cut -d: -f4,5) | ||
| IFS='|' read -ra vars <<< "${HRRR_VARS}" | ||
| for var in "${vars[@]}"; do | ||
| if ! echo "${metadata}" | grep -E -q "^${var}(:| )?"; then | ||
| return 1 | ||
| fi | ||
| done | ||
| return 0 | ||
| } | ||
| export -f check_file_is_valid |
There was a problem hiding this comment.
Here is one where we need to use check_file_existence
There was a problem hiding this comment.
Thank you for pointing to these!
| # Write index if it does not exist yet | ||
| if [[ -n "${file_idx_data}" ]]; then | ||
| echo "${file_idx_data}" > "${file}.idx" | ||
| fi |
There was a problem hiding this comment.
How about doing this on line #136 and then using that as the basis for "grepping". If we have a miss, we will delete it again on line #146
I know that creating this index is quick and this way we would still only do it once
Addresses #34 . The idea here is to fold it into the existing loop we do which cycles through all of the data sources if there is no data found. Suggested change is if not all
HRRR_VARSare found, we delete the file and will automatically download from another source.